When a user performs a search on a commercial search engine and then clicks on the results, the commercial search engine may gather information about which results were presented to the user and about the particular results the user clicked. The commercial search engine operators may then use this information to evaluate the quality of the search, to improve the search, and to perform machine learning to improve the quality of the search results.
For example, if a commercial search engine has a new algorithm for determining search results for a search query, the commercial search engine may present results from the new algorithm, and compare the click rate of the results from the new algorithm to the click rate of the results from the old algorithm. A higher click rate on results from the new algorithm suggests that it is superior. Examples of such approaches may be found in J. Boyan, D. Freitag, and T. Joachims, A Machine Learning Architecture for Optimizing Web Search Engines, Proceedings of the AAA1 Workshop on Internet Based Information Systems, 1996.
Unfortunately, users cannot always determine if a result is “good,” or how good it is, without clicking on it. Users may click on a result by mistake. Moreover, users sometime click on results that are not superior, and in many cases, inferior, to other results. For example, the title and snippets associated with some search results may mislead users and result in an artificially-high click rate. The fact that users are misled may be accidental or deliberate. For example, some webmasters adjust their pages to make them appear artificially good in search result lists, in order to draw additional traffic to their sites. For example, some web sites extract information from a search query and insert the information into the result title or snippet, making the result appear as if it closely matches the search query. Thus, evaluating the quality of search results based solely on which results in a result set are selected by users (or “clicked on”) may not yield an effective evaluation.
Some search engines associated with electronic-commerce sites have devised methods of tracking user behavior in an attempt to more accurately rank search results. For example, some commercial shopping sites track a user's behavior by determining whether they buy particular products when they are shown to the user. If the purchase rate increases, the ranking of the result is increased. This approach, too, has limited effectiveness. For example, this approach may not be available, to a search engine not associated with an electronic-commerce site, and access to such data may not be available. Moreover, for many searches, a purchase may not constitute a good indicator of user satisfaction.